Regularization for Neural Networks

Research into regularization techniques is motivated by the tendency of neural networks to to learn the specifics of the dataset it was trained on rather than learning general features that are applicable to unseen data. This is known as overfitting. The goal of any supervised machine learning task is to approximate a function that maps inputs to outputs, given a dataset of examples and labels. An important assumption is that models are trained on datasets which are representative of the true distribution of the data and the target function to be approximated. However, almost all data is noisy and contains some random deviations from the true distribution. Given this fact, approximating the function represented by the training data very precisely is undesirable. The algorithm will learn the noise in the training dataset and will be unlikely to perform well when applied to unseen data. Regularizing neural networks helps them to learn the true function and ignore the noise.